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Abstract 

The  primary  focus  of  this  research  effort  was  the  tracking  of  eye 
movements  during  complex  cognitive  tasks.  8  volunteers 
performed  a  visual  tracking  task  alone  and  a  combination  of  the 
visual  task  and  the  auditory  Paced  Serial  Addition  Task  (PASAT). 
Results  showed  a  reduction  in  range  on  the  order  of  50%  for  eye 
movements  and  an  increase  in  variability  of  vergence  eye 
movements  during  the  dual-task.  This  change  can  be 
characterized  as  visual  tunneling. 

1.  Objective  and  Background 

In  optimization  of  the  design  of  new  systems  and  adaptation  of 
existing  teehnologies,  the  aeeurate  measure  and  predietion  of 
human  operator  workload  ean  be  eritieal.  Within  the  eontext  of  a 
eomplex  human-maehine  system,  there  is  the  environment,  task, 
and  operator  state.  It  is  the  latter,  operator  state,  whieh  is  the 
foeus  of  this  study. 

Visual  attention  has  inereased  in  importanee  as  the  systems  into 
whieh  humans  are  being  integrated  beeome  more  visually 
eomplex.  The  ability  of  humans  to  attend  to  a  variety  of  stimuli  in 
these  eomplex  systems  is  often  the  limiting  faetor  in  overall 
system  performanee.  Eye  movement  behaviour  ean  now  be 
deteeted  by  stand-off  video  eameras.  Thus  a  system  that  eould 
reliably  deteet  eye  movement  behaviour  patterns  related  to 
operator  states  sueh  as  fatigue,  loss  of  attention  or  workload 
would  be  then  able  to  alter  the  task  presentation  to  optimize 
operator-system  performanee.  In  this  pilot  study  we  sought  to 
extend  the  proeess  of  understanding,  quantifying  and  modelling 
eye  movement  behaviours  that  are  indieative  of  physiologieal  and 
eognitive  states. 

Human  eye  traeking  teehnology  has  experieneed  signifieant 
improvement  in  reeent  years  in  sueh  eritieal  areas  as  sampling. 
Further,  redueed  eosts  and  improved  reliability  of  systems  makes 
it  feasible  to  monitor  both  eyes  and  thus  provide  additional 
parameters  to  measure,  a  seeond  eye  and  its  eorrelation  with  the 
first  eye.  To  date,  many  eye  aetivity  behaviours  have  been 
eorrelated  with  visual  and  eognitive  demands  of  various  tasks. 
What  is  laeking  in  the  eurrent  body  of  literature  are  a  variety  of 
real  time  models  of  human  operator  state  that  ean  be  integrated 
into  system  funetions.  In  the  present  study  we  wished  to  examine 
measures  of  operator  workload  as  indieated  by  oeulomotor 
aetivity.  We  ehose  to  eombine  a  visual  task  (traeking  an  objeet) 
with  an  auditory  task.  A  priori,  the  auditory  task  should  have  no 


direet  requirement  for  ehange  of  oeulomotor  behaviour  while  it  is 
being  performed.  Thus  any  ehanges  in  behaviour  may  potentially 
be  aseribed  to  the  inerease  in  workload  when  going  from  visual 
alone  to  visual  and  auditory. 

The  partieipants  were  8  volunteers  who,  while  wearing  the  head- 
mounted  eye  traeker,  performed  a  visual  traeking  task  and  a 
eombination  of  the  visual  traeking  task  and  auditory  task. 

A  simulated  sereen  shot  of  the  visual  traeking  task  is  shown  in 
Figure  1.  The  subjeet  uses  a  joystiek  to  eenter  in  two  dimensions 
a  visual  target  that  is  being  eonstantly  perturbed.  This  task  is  a 
reasonable  proxy  for  simultaneous  lane  keeping  and  speed 
maintenanee  during  driving. 

The  seeond  task  is  an  auditory  task  -  a  version  of  the  Paeed 
Auditory  Serial  Addition  Test  (PASAT).  In  this  task  the  subjeet 
hears  a  series  of  numbers  and  must  then  add  eaeh  number  to  the 
previous  number  and  state  the  sum.  Task  diffieulty  is  a  funetion 
of  number  presentation  speed.  It  ean  demand  total  attention  from 
the  subjeet  at  moderately  fast  speeds  for  sueeessful  performanee. 

We  reeorded  eye  movements  through  high-bandwidth,  binoeular 
traeking.  The  eye  traeking  system  utilized  in  this  study  was  the 
EyeFink  II,  from  SR  Researeh  Ftd,  Mississauga,  Canada.  Video- 
based  eye  traeking  systems  have  long  been  aeknowledged  as  the 
simplest  to  set  up  and  operate.  EyeFink  II  has  a  high  resolution 
(noise-limited  at  <0.01°)  and  fast  data  rate  (500  samples  per 
seeond).  This  exeeptional  data  quality  results  in  very  low  veloeity 
noise,  making  the  EyeFink  II  ideal  for  saeeade  analysis  and 
smooth  pursuit  studies.  On-line  gaze  position  data  is  available 
with  delays  as  low  as  3  milliseeonds,  making  the  system  ideal  for 
gaze-eontingent  display  applieations.  In  addition,  on-line  data 
parsing  oeeurs,  making  eye  events  sueh  as  saeeade,  fixation,  and 
blink  available  within  25  ms  to  the  display  eomputer. 

We  tested  subjeets  performing  eaeh  task  independently  and  both 
tasks  simultaneously.  Our  initial  analysis  shows  it  may  be 
possible  to  diseriminate  between  a  visual  and  an  auditory  foeus 
based  on  eye  movements  for  independent  task  performanee. 

Subjeets  aneedotally  reported  that  they  felt  they  were  doing  the 
visual  traeking  task  automatieally  and  without  attention. 
However,  we  found  differenees  in  eye  movement  behaviour  when 
performing  the  visual  traeking  task  alone  and  performing  both 
tasks  together.  In  partieular,  depth  of  field  ehanges  indieated  that 
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the  subject  was  occasionally  not  fixated  on  the  screen  during  the 
combined  task. 


Figure  1.  Driving  analog  task.  Screen  shot  (simulated)  of  visual 
tracking  task 

2.  Results 

We  have  plotted  the  eye  movements  of  a  subject  during  the 
three  tasks:  the  Paced  Auditory  Serial  Addition  Task 
(PAS AT)  in  Figure  2,  the  visual  tracking  task  in  Figure  3, 
and  the  combined  PAS  AT  +  tracking  task  in  Figure  4. 
Horizontal  and  vertical  axes  are  the  X  coordinate  and  the  Y 
coordinate  of  the  position  of  the  left  eye  (in  red)  and  the 
right  eye  (in  blue)  gaze  directions  on  the  monitor  screen  in 
front  of  the  subject,  measured  in  pixels.  The  vertical  axis  is 
the  time  from  the  start  to  the  finish  of  the  tasks,  in 
milliseconds. 


Eye  movement  dunng  the  aiMftory  PASAT  task 


Figure  2  Eye  movement  during  the  auditory  Paced  Auditory  Serial 
Addition  Task  (PASAT)  task.  Horizontal  axes  are  the  X 
coordinate  and  the  Y  coordinate,  respectively,  of  the  position  of 
the  left  eye  (in  red)  and  the  right  eye  (in  blue)  gaze  directions  on 
the  monitor  screen  in  front  of  the  subject,  measured  in  pixels.  The 
vertical  axis  is  the  time  from  the  start  to  the  finish  of  the  tasks,  in 
milliseconds. 


Figure  3.  Eye  movement  during  the  visual  tracking  task. 
Horizontal  axes  are  the  X  coordinate  and  the  Y  coordinate 
respectively,  of  the  position  of  the  left  eye  (in  red)  and  the  right 
eye  (in  blue)  gaze  directions  on  the  monitor  screen  in  front  of  the 
subject,  measured  in  pixels.  The  vertical  axis  is  the  time  from  the 
start  to  the  finish  of  the  tasks,  in  milliseconds. 


Figure  4.  Eye  movement  during  the  combined  PASAT-and- 
tracking  task.  Horizontal  axes  are  the  X  coordinate  and  the  Y 
coordinate  respectively,  of  the  position  of  the  left  eye  (in  red)  and 
the  right  eye  (in  blue)  gaze  directions  on  the  monitor  screen  in 
front  of  the  subject,  measured  in  pixels.  The  vertical  axis  is  the 
time  from  the  start  to  the  finish  of  the  tasks,  in  milliseconds. 


Comparing  Figures  2  and  3  for  the  two  single  tasks  we  see  that 
during  the  auditory  task  the  movements  of  the  two  eyes  hover  in  a 
region  of  the  screen  without  fixating  upon  any  particular  location. 
Also  in  figure  2.,  the  movements  of  the  two  eyes  are  quite  random 
and  completely  superimpose  each  other. 

In  comparison,  during  the  visual  task  (figure  3)  the  total  extent  of 
excursion  is  smaller  (by  about  half)  than  for  the  auditory  task 
(note  change  in  scale).  As  in  the  auditory  task,  the  eyes  also 
remain  well  coordinated.  It  is  worth  noting,  however,  that  in  the 
auditory  task,  even  though  the  eyes  are  free  to  roam  anywhere 
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since  there  is  nothing  of  relevance  to  task  performance  in  the 
visual  field,  they  tend  to  hover  in  a  rather  small  region  of  the 
visual  field  (without  fixating  at  any  point  in  particular.)  This 
seems  consistent  with  the  finding  that  when  people  are  paying 
attention  (in  this  case,  to  an  auditory/cognitive  task)  their  gaze 
narrows  down,  and  consequently,  so  does  their  ability  to  survey 
their  visual  environment  (visual  tunneling). 

During  the  combined  visual/auditory  task  (figure  4.)  shows  a 
dramatically  reduced  extent  of  eye  movements  and  an  apparent 
increase  in  the  disconjugacy  of  the  two  eyes. 

The  following  plots  (Figure  5)  show  the  spatial  range  of  the  eye 
movements  classified  as  fixations  or  smooth  pursuits  on  the 
screen  in  front  of  a  representative  subject  (subject  5)  for  the  two 
tasks:  tracking  and  combined  (for  the  two  numeral  presentation 
ISIs  of  240  and  200  ms.)  Left  eye  movements  are  shown  in  blue, 
right  eye  movements,  in  red. 

Subject  5:  Range  of  fixations/smooth  pursuits 
Fixations/smooth  pursuits  during  tracking  task 
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Figure  5.  Plot  showing  the  extent  in  screen  coordinates  of  the  eye 
movement  during  fixations  and  smooth  pursuits  for  subject  5.  Left 
eye  movements  are  shown  in  blue,  right  eye  movements,  in  red. 

All  eight  subjects  showed  a  reduction  in  the  spatial  extent  of 
movements.  To  quantify  the  reduction,  standard  deviations  of  the 
spatial  extent  were  calculated. 

Right  Eye 


Figure  6.  (above)  Graph  showing  the  standard  deviation  of  the 
range  of  fixation/smooth  pursuit  eye  movements  (in  pixels)  for 
subjects  S1-S8.  Blue  bars  show  the  data  for  the  tracking  task,  red, 
for  the  combined  task  with  presentation  ISI  of  240  ms,  and 
yellow,  for  the  combined  task  with  presentation  ISI  of  200  ms. 

Subjects  showed  on  average  at  least  50%  reduction  in  standard 
deviation  of  the  spatial  extent.  Except  for  subjects  2  and  3  there  is 
more  reduction  in  standard  deviation  when  the  numeral 
presentation  ISI  is  decreased. 

The  following  graph  (Figure  7)  are  the  plots  of  the  spatial  range  of 
saccadic  eye  movements,  again  for  subject  5,  and  the  two  tasks: 
tracking  and  combined  (presentation  ISI  =  240  and  200  ms.).  As 
before,  left  eye  movements  are  in  blue,  right  eye  ones  in  red. 

Subject  5:  Range  of  saccades 
saccadic  eye  dunng  tracking  task 
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Figure  7.  Plots  showing  the  extent  in  screen  coordinates  of  the  eye 
movement  during  saccades  for  subject  5.  Left  eye  movements  are 
shown  in  blue,  right  eye  movements,  in  red. 

Clearly  the  same  results  hold  for  saccades  as  for  fixations/smooth 
pursuits:  the  range  over  which  the  eyes  roam  reduces  by  about 
50%  or  more  as  the  PAS  AT  task  is  added  to  the  tracking  task. 

The  consistent  reduction  in  spatial  range  of  both  kinds  of  eye 
movements  as  the  PASAT  is  added  to  the  tracking  task  can  be 
characterized  as  “visual  tunneling”.  A  possible  explanation  for  the 
effect  is  as  follows:  The  cognitive  effect  of  adding  the  PASAT 
task  to  the  tracking  task  is  an  increase  in  cognitive  load,  and 
therefore,  an  increase  in  attentional  resources  demanded  of  the 
subjects.  Further,  the  greater  the  presentation  speed  in  the  PASAT 
task,  there  will  be  an  additional  increase  in  attentional  demand. 
The  increased  attentional  demand  results  in  focusing  of  visual 
attention  to  a  smaller  portion  of  the  visual  field,  that  portion,  in 
fact,  that  is  most  essential  to  the  performance  of  the  visual  task. 
Excursions  to  other  areas  of  potential  visual  interest  are 
minimized.  This  reduction  of  movement  would  come  at  the  price 
of  reducing  the  ability  to  detect  peripheral  items  of  interest. 
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Figure  8.  Vergence  angle  (angle  between  right  and  left  eyes)  on 
traeking  alone  (upper)  and  Traeking  +  auditory  task  (ISI  240ms, 
lower  left  and  ISI  200ms,  lower  right) 


The  angle  between  the  two  eyes  as  they  move  is  ealled  the 
vergenee  angle.  This  angle  inereases  as  objeets  are  eloser  to  the 
eyes.  Vergenee  eontrol  is  a  different  oeulomotor  eommand  system 
than  other  eye  movements.  Figure  8  (above)  shows  that  there  is 
greater  variability  in  vergenee  angle  as  the  dual  tasks  are 
performed.  This  partieularly  oeeurs  during  saeeadie  refixations, 
whieh  appear  as  “spikes”  in  the  figure.  Figure  9  shows  the 
inerease  in  standard  deviation  of  the  vergenee  angle  as  the 
auditory  tasks  are  added. 


Figure  9  (below):  Chart  showing  the  standard  deviation  of  the 
vergenee  (difference  between  the  left  and  right  eye  x-coordinates, 
in  pixels)  for  subjeets  S1-S8.  Blue  bars  show  the  data  for  the 
traeking  task,  red,  for  the  eombined  task  with  presentation  ISI  of 
240  ms,  and  yellow,  for  the  eombined  task  with  presentation  ISI 
of  200  ms. 


3.  Impact 

Changes  in  oeulomotor  behaviour  appear  to  signal  a  ehange  in 
operator  workload.  An  important  question  for  further  researeh  is 
to  determine  if  the  ehange s  are  task  speeifie  or  are  more 
fundamentally  related  to  the  workload  eondition.  For  example, 
visual  tunneling  of  attention  might  not  be  all  there  is  to  explaining 
reduetion  in  amplitude  of  smooth  pursuits.  It  is  eoneeivable  that 
with  the  periodie  engagement  of  attention  by  the  PASAT  task,  the 
ability  of  the  visual  system  to  smooth-pursue  the  eursor  is 
redueed.  The  task  of  traeking  the  eursor  in  this  situation  is 
aeeomplished  mostly  by  saeeades.  At  moments  of  loss  of  attention 
to  the  PASAT  task,  the  eyes  either  fixate  or  may  roam  at  random 
showing  high  vergenee  variability. 

Although  previous  studies  have  demonstrated  general  eye  aetivity 
ehanges  with  shifts  in  task  demand,  few  studies  have  attempted  to 
eombine  multiple  eye  measures  or  utilize  suffieiently  short  (sub  1 
minute)  integration  times.  Both  of  these  analytie  methods  appear 
to  be  required  in  a  system  that  would  eontribute  to  real-time 
workload  estimation.  Reeent  studies  by  Van  Orden,  Limbert, 
Makeig,  and  Jung  [1]  and  Van  Orden,  Jung,  and  Makeig  [2],  on 
eye  aetivity  eorrelates  of  fatigue  sueeessfully  estimated 
performanee  in  a  sustained  visual  traeking  task  by  eombining  60-s 
moving-window  estimates  of  eye  aetivity.  In  our  dual-task 
eondition  a  similar  signal-proeessing  approaeh  eould  be  used  to 
analyze  eye  data  during  a  task  in  whieh  memory  and  visual 
aetivity  demands  are  varied  over  time.  The  aeeuraey  of  models 
eombining  eye  measures  in  predieting  moment- to-moment 
fluetuations  in  task  demand  eould  be  assessed  in  future  work. 
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